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The Data Acquisition System Based on PMC Bus 
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High energy physics experiments in KEK/ Japan rush into over kHz trigger stage. Thus, we need a successor of 
the data acquisition(DAQ) system that replaces the CAMAC or FASTBUS systems. To meet these needs, we 
have developed a widely usable high-density DAQ system which includes a crate, base-board, daughter cards for 
front-end A/D or T/D conversion, and back-end communication base-board for data transfer and timing control. 
The size of the crate is for the 9U Euro-cards with the standard VME32 bus and extension connectors for power 
supply. The base-board comprises of a local bus with the sequencer connected to the front-end daughter cards 
via event buffering FIFOs, and the standard PMC (PCI mezzanine card) bus to be set a PMC processor unit 
to reduce data size from the front-end daughter cards. A data transfer module, which is connected to the event 
building system, and a trigger control unit, which communicates with the central timing controller are installed 
on the back-end communication card connected to the rear end of the base-board. We describe the design of 
this DAQ system and evaluate the performance of it. 



1. Introduction 

The data acquisition (DAQ) system of the next gen- 
eration experiments proposed in KEK/ Japan must 
operate with trigger rates significantly higher than the 
currently on-going experiments. Numbers of experi- 
ments are planned at a high intensity 50 GeV pro- 
ton accelerator project (J-PARC Q). The KEKB B- 
factory also plans to upgrade to a luminosity ten times 
higher C = 10 35 cm- 2 s- 1 (Super KEKB 2]). The trig- 
ger rate of these experiments are estimated as 1 kHz 
~ 10 kHz for the J-PARC experiment and -10 kHz 
for upgraded the KEKB experiment. In the above 
ranges, conventional DAQ systems based on CAMAC, 
Fastbus, or VME are not appropriate for our needs: 
CAMAC does not allow high density channels because 
of its small board size; Fastbus is commercially dying 
out; and VME does not supply adequate power for 
analog signals. Because there is no suitable (de-facto) 
standard system capable of working at such a trig- 
ger rate, we developed a DAQ platform which may 
have a wide range of uses. The first application is 
the DAQ system for experiments at J-PARC and the 
Super KEKB experiment. 



2. Design overview 

We design the DAQ platform with the following fea- 
tures: 

• Front-end Buffering: 
Signals from detectors are buffered in the front- 



end while awaiting a trigger decision and the 
timing signals for data transfer are uniformly 
distributed to the back-end. This is effective at 
reducing the dead-time. 

• Trigger/busy handshake: 

The Trigger /busy handshake works at a trigger 
rate of up to 10 kHz. 

• Front-end data reduction: 

Large amounts of data generated by Flash 
ADC(FADC), can be reduced on the board. 

• Data transporting by data link: 

Data are transferred downstream to the DAQ 
system via a standard data link such as a Fast 
Ethernet. 

• Modular system: 

The system is composed of modules for maxi- 
mum flexibility, allowing choice of digitization 
scheme, processor and data link. Being modu- 
lar also makes it easier to upgrade, for example, 
the processor or data links. 

• Wide scalability: 

The same DAQ system can be used not just for 
small-scale tests but also for large-scale experi- 
ments such as the Super KEKB experiment 

• Cost effectiveness 

We developed a 9U-size power crate with a 9U-size 
main board and a 6U-size rear board as the base ele- 
ments. The power crate has a standard VME32 bus 
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Figure 1: KEK-VME crate with a read-out module 



and an extension connector for the power supply. The 
main board is equipped with a PCI mezzanine card 
(PMC) bus so that the fastest processor available can 
be used. The main board has three PMC slots and 
four front-end daughter card slots. Each of the PMC 
slots and each of the daughter card slots have a FIFO 
buffer for efficient data handling. The PMC buses 
run from the main board to the rear board. The 
rear board has two PMC slots and a special slot for 
the trigger distribution card. The front-end function, 
such as FADC or pipeline TDC, can be installed as 
front-end daughter cards similar in size to PMC cards. 
The event signals digitized by the front-end cards are 
buffered in FIFOs, transferred to the processor via the 
PMC bus, reduced by the processor, and then sent to 
the event-builder system through the data transfer in- 
terface via the PMC bus. 

The trigger information is received by the trigger 
card on the rear board and distributed to each front- 
end card. Figure [I] shows the crate and a read-out 
module used for the test. 

Details of the DAQ platform components are de- 
scribed in the following sections. 



Table I KEK-VME JO connector pin-out definition 



Pos. 


z 


a 


b 


c 


d 


e 


f 


1 


GND 


GND 


GND 


GND 


GND 


GND 


GND 


2 


GND 


GND 


GND 


GND 


GND 


GND 


GND 


3 


GND 


GND 


GND 


GND 


GND 


GND 


GND 


4 


GND 


+3.3V 


+3.3V +3.3V +3.3V +3.3V 


GND 


5 


GND 


+3.3V 


+3.3V +3.3V +3.3V +3.3V 


GND 


6 


GND 


+3.3V 


+3.3V +3.3V +3.3V +3.3V 


GND 


7 


GND 


+3.3V 


+3.3V 


GND 


GND 


GND 


GND 


8 


GND 


GND 


GND 


GND 


GND 


GND 


GND 


9 


GND 


GND 


GND 


GND 


GND 


GND 


GND 


10 


GND 


GND 


GND 


GND 


-3.3V 


-3.3V 


GND 


11 


GND 


-3.3V 


-3.3V 


-3.3V 


-3.3V 


-3.3V 


GND 


12 


GND 


-3.3V 


-3.3V 


-3.3V 


-3.3V 


-3.3V 


GND 


13 


GND 


GND 


GND 


GND 


GND 


GND 


GND 


14 


GND 


-5V 


-5V 


-5V 


-5V 


-5V 


GND 


15 


GND 


GND 


GND 


GND 


GND 


GND 


GND 


16 


GND 


S1+ 


Sl- 


GND 


S2+ 


S2- 


GND 


17 


GND 


S3+ 


S3- 


GND 


S4+ 


S4- 


GND 


18 


GND 


S5+ 


S5- 


GND 


S6+ 


S6- 


GND 


19 


GND 


S7+ 


S7- 


GND 


S8+ 


S8- 


GND 




KEK-VME Back Plans (Shiiradu Aug.13,2002) 



Figure 2: KEK-VME crate and it's part of back-plane 



3. KEK-VME crate, an extension of 
standard VME 

We chose a 9U Euro card with a VME32 bus as the 
base of our system because these are widely used and 
are cost effective. However, the default VME system 
cannot provide sufficient current needed for digitiza- 
tion analog devices. Moreover, the standard VME 
power supply is too noisy for delicate handling of the 
analog signals. Therefore, we added a dedicated con- 
nector as JO for the power supply, to avoid any con- 
cern and to simplify the power treatment. Supplied 
voltages via the JO connector are -5.0 V and ±3.3 
V. Maximum currents delivered by JO connector are 
100A for -5.0 V, 320 A for +3.3 V and 200 A for -3.3 V. 
KEK-VME JO connector is used as same as VME64x 



JO connector. The definition of pins are shown in Ta- 
ble [H Eight pairs of the differential-signal bus line 
for the 100-Mbps LVDS are assigned as Sl±, S2±, 
S3±, S4±, S5±, S6±, S7± and S8± in the JO. Its 
line impedance of the differential-signal bus line is 100 
Ohm. 

Moreover, we are also developing a low noise and 
cost-effective power supply. 



4. Read-out module 

FADC and pipeline TDC generate large amounts of 
data. To reduce the data at an early stage, a pipeline 
buffer and a processor were installed on the board. 

The read-out module meets the following require- 
ments: 
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Figure 3: A schematic view of read-out module 

• A upgradeable processor: because processors are 
evolving rapidly. 

• A changeable data transfer interface: because 
this technology is also evolving rapidly and ex- 
perimental setups call for different requirements. 

To satisfy these requirements, we have developed a 
new DAQ system equipped with a PMC for the digital 
part of the read-out module. 

The system consists of four components: two types 
of base boards, a front-end mezzanine card, and a 
PMC . One of the base boards is named COPPER 
(COmmon Pipelined Platform for Electronics Read- 
out). This has four slots for the front-end mezzanine 
cards on half of the front side, a PMC slot for the 
processor, and two universal PMC slots on half of the 
back side. The other is a rear board named SPIGOT 
(Sequencing Pipeline Interface with Global Oversight 
and Timing), which has two universal PMC slots and a 
slot for a trigger module. A front-end mezzanine card 
named FINESSE (Front-end INstrumentation Entity 
for Sub-detector Specific Electronics) is a daughter 
card with A/D converting function. Various types of 
FINESSE have been made, such as multiple TDC and 
FADC. 

4.1. PMC(PCI Mezzanine Card) 

The PMC, denned by IEEE1386.1 4J, separates the 
processor unit and the read-out module. Its func- 
tions and electrical characteristics are the same as 
a PCI, but it measures only 74 mm x 149 mm. It 
can be used with VME, VME64, VME64x, Compact 
PCI, etc. Several modules are available at reasonable 
prices, as the technology involved is the same as that 
for PCIs used in common PCs. 

4.2. Base board 

One of the base boards is a 9U-size main base board 
designated COPPER (COmmon Pipelined Platform 
for Electronics Readout). The other is a 6U-size rear 




Figure 4: The COPPER and SPIGOT base boards 



base board designated SPIGOT (Sequencing Pipeline 
Interface with Global Oversight and Timing). The 
boards are shown in Figure 21 

The front half of the COPPER is for signal digitiz- 
ing and the back half is for data processing. The front 
has four slots for front-end mezzanine cards, which are 
described below. The back has a PMC slot for the pro- 
cessor and two universal PMC slots. A bus bridge, us- 
ing PLX technology incorporated in the PCI 9054(3, 
connects the front local bus and the back PMC bus. 

Eight 256KB FIFOs receive data from the mezza- 
nine cards which are passed to the memory on the 
processor board by a sequencer on the local bus and 
the PLX PCI-9054. 

The SPIGOT is at the back and is connected to 
the COPPER via a PCI bus and trigger information 
lines. The SPIGOT has a trigger information module 
and a PMC card for a data transfer interface such as 
Ethernet or a serial link(IEEE1394, etc). 

4.3. Processor PMC 

As the processor PMC card, we use a Radisys EPC- 
63150. 

The specification of the processor are:. 

• 800 MHz Pentium III-M Processor 

• up to 512 MB PC-133 compliant SD-RAM with 
ECC 

• 10/100 BaseT Ethernet port 

• On-board CompactFlash socket 

• 32-bit 33/66 MHz PCI bus interface 

We chose PC architecture for front-end data pro- 
cessing, because PC architecture has the following 
merits: 

• It is familiar to many people because it has the 
same architecture as the common PC. 
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Figure 6: FINESSE-jig, a front-end daughter card for 
testing system 



• PCs can run operating systems rich in function. 
Therefore we can develop applications easily and 
drive many complex devices (such as Ethernet 
TCP/IP). 

• PC are common throughout the world, and fast, 
yet cheap, processors are readily available. 

• A large amount of knowledge is openly available. 

Processor PMC card EPC-6315 shown in Figure [5] 
We use Linux 2.4 on a on-board processor. Linux 

is a most popular development environments in high 

energy physics experiments. 

4.4. FINESSE, Front-end daughter card 

The FINESSE (Front-end INstrumentation En- 
tity for Sub-detector Specific Electronics) front-end 
daughter card for testing the system was developed 
first, instead of a real FADC or pipeline TDC. It is 
called a FINESSE-jig and has FIFO logic and trig- 
ger/busy handshaking logic integrated on it. It gener- 
ates the simulated data, and stores it into local FIFO 
and exports it to the other FIFO on the base board 
upon receiving a trigger. FINESSE-jig is shown in 
Figure 

FINESSE is being further developed. A recently de- 
veloped FINESSE is the FINESSE AMT2 which is a 
TMC (time memory cell) Q based pipeline TDC card, 
has an AMT2 chip. The AMT2 (ATLAS Muon TDC 
2)0 chip is the latest TMC chip developed for AT- 
LAS muon chamber. Its specifications are shown in 
Table El 

FINESSE AMT2 is shown in Figured 



Table II Specifications of FINESSE AMT2 



TDC 


AMT2 


Input 


24 ch LVDS 


Time resolution 


0.78 ns/bit (at using 40 MHz clock) 


Trigger buffer depth 


8 words 




Figure 7: FINESSE AMT2, a pipeline TDC front-end 
daughter card 

Another recently developed FINESSE is the FI- 
NESSE FADC which uses for four FADC chips, Ana- 
log Devices incorporated AD9235-20. Its specifica- 
tions are shown in Table UTTl 



5. Performance and stability 

We have tested the on-board PCI bus and local bus 
system for data transfer speed, stability over long pe- 
riods of continuous use, and thermal stability. 

We measured the speed of data transfer from the FI- 
NESSE to the processor main memory in DMA mode, 
using the FINESSE- jig. The measured transfer speed 
without overhead was 125 MB/sec. This result shows 
that it works with 95% of the 32bit 33MHz PCI bus 
performance. We also measured the transfer speed 
including the overhead of the sequencer on the local 
bus. The speed was 80 MB/sec when the event size 
was 200 bytes. 

To test for module reliability, we ran the system 
for 77 hours continuously. To check for error, all the 
transferred data were compared with data generated 
by FINESSE-jig. 

To test the thermal stability, we placed the read- 
out module in a thermostatted oven and changed the 
temperature as shown in Figure 

Data generated by FINESSE-jig were read by FI- 
NESSE FIFO, transferred by the PLX PCI-9054 DMA 



Table III Specifications of FINESSE FADC 



ADC 


Analog Devices AD9235-20 


Number of channel 


8 ch 


Resolution 


12 bit 


Maximum sampling clock 


20 MHz 
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Thermostatic 
Oven 




Receiver PC 



Figure 8: Setup for stability tests 




Time 

Figure 9: Temperature variation in the oven during the 
stability test 

via a local bus and a PCI bus to the processor main 
memory, and then sent using a LAN card via a 100 
BaseT network to another computer outside the oven. 
In order to simulate actual use, we made the setup as 
shown in Figure |H1 Data generated by FINES SE-jig 
were stored into FINESSE FIFO, transferred by the 
PLX PCI-9054 DMA via a local bus and a PCI bus 
to the processor main memory, and then sent to an- 
other computer outside the oven through a LAN card. 
The test was performed at average trigger frequency 
10 Hz. 

No error occurred in the test. The complete system 
worked correctly. 



6. Summary 

We have developed a widely usable high-density 
DAQ system equipped with a PMC bus as the internal 
bus. It has data buffers just behind the digitizers to 
reduce DAQ dead-time. On-board data are reduced 



by a common PC architecture. The system is modular 
and so the processor and the data transfer interface 
are upgradeable. The system is flexible and scalable 
and may be applied to a wide range of experiments be- 
cause of its modular structure and data link interface 
for data transfer. 

The performance of data transfer from FIFOs to 
processor main memory is 125 MB/sec without errors. 

We are developing a refined version of the system 
with digitizing front-end daughter cards, and a trigger 
handling and distribution system. 
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